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Abstract. This paper examines a systematic method to con- 
struct a pair of (inter-related) root systems for arbitrary Cox- 
eter groups from a class of non-standard geometric representations. 
This method can be employed to construct generalizations of root 
systems for a large family of linear groups generated by involu- 
tions. We then give a characterization of Coxeter groups, among 
these groups, in terms of such paired root systems. Furthermore, 
we use this method to construct and study the paired root systems 
for reflection subgroups of Coxeter groups. 



1. Introduction 

A Coxeter group W is an abstract group generated by a set of invo- 
lutions i?, called its Coxeter generators, subject only to certain braid 
relations. Despite the simplicity of this definition, there is a rich theory 
for Coxeter groups with non-trivial applications in a multitude of areas 
of mathematics and physics. When studying Coxeter groups, one of 
the most powerful tools we have at our disposal is the notion of root 
systems. In classical literature (O Ch.V, §4] or [211 §5-3-5.4], for exam- 
ple), the root system of a Coxeter group is a geometric construction 
arising from the Tits representation of W . The Tits representation of 
W is an embedding of W into the orthogonal group of a certain bilinear 
form on a suitably chosen vector space V subject to the requirement 
that the VU-conjugates of elements of R are mapped to reflections with 
respect to certain hyperplanes in V . In the case that W is finite, V is 
Euclidean (of dimension equal to the cardinality of i?), and the root 
system of W simply consists of representative normal vectors for these 
hyperplanes. Those elements of the root system corresponding to the 
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elements of R are known as simple roots, and in most classical liter- 
ature ([21 Ch.V, §4] and §5.3], for example), the simple roots are 
linearly independent. 

Similar constructions of root systems can be extended to infinite 
Coxeter groups and Kac-Moody Lie algebras. However, the actual con- 
structions of root systems differ depending whether the root systems 
are associated to Kac-Moody Lie algebras or infinite Coxeter groups. 
As discussed in the introduction of [17J, while all definitions of root 
systems are related to a given bilinear form, the actual bilinear forms 
considered in the case of Kac-Moody Lie algebras are different from 
the ones in Coxeter groups. Furthermore, it is well known ([H] and (TUl 
Ch.3]) that within an arbitrary Coxeter group W, all of its reflection 
subgroups are themselves Coxeter groups, but in the literature ( [TU] 
or [llj, for example), the construction of the root systems correspond- 
ing to such reflections subgroups as subsets of the root system of W 
requires special care. In particular, since a proper reflection subgroup 
may have strictly more Coxeter generators than the over-group (as seen 
in [171 Example 5.1]), the equivalent of the simple roots in these root 
systems need not be linearly independent, making the overall theory 
of root systems and root bases somewhat less uniform. Consequently, 
it seems profitable to develop a universal method for constructing root 
systems that is applicable to arbitrary Coxeter groups and their reflec- 
tion subgroups, as well as to objects like Kac-Moody Lie algebras (in 
fact, to all groups with a so-called root group datum, as surveyed in 
i). 

In [25], [7], [8] and [6], a number of more general notions of root 
systems have been studied. In these approaches, a pair of root systems 
are constructed in two vector spaces which are essentially algebraic 
duals of each other (apart from [2S], the two vector spaces involved are 
explicitly required to be algebraic duals of each other, whereas in [22] 
the two vector spaces are linked by a non-degenerate bilinear pairing 
satisfying certain integrality conditions). 

Recently, an approach taken in [12] and [H] generalizes those of [25], 
[7], [8] and [6]. In this approach, for an arbitrary Coxeter group, a 
pair of root systems are constructed in two vector spaces linked only 
by a bilinear pairing which does not require the non- degeneracy and 
integrality conditions of [2B] . This particular approach allows more ab- 
stract geometry in the two root systems to take place (for example, the 
two representation spaces need not be algebraic duals of each other), 
whilst providing a unified theory of root systems, especially with re- 
spect to reflection subgroups (this last point is to be established in 
Section [3] of this present paper). 

In this paper, we present a few results demonstrating the "univer- 
salness" of the notion of root systems in [12] and [H]. In fact, this 
new approach applies to a large family of linear groups generated by 
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involutions, and one of the key results of this paper (Theorem 12. 8p 
shows that these groups are Coxeter groups only if the corresponding 
root systems decompose as disjoint unions of those roots generalizing 
the classical concept of positive roots and those roots generalizing the 
classical concept of negative roots. In fact, this result provides an alter- 
native characterization for Coxeter groups, since it is well known that 
for any Coxeter group we may construct a root system that decom- 
poses in the same way. This alternative characterization is implicitly 
suggested in the work of Matthew Dyer ( [10] ) , and we are very grateful 
to Prof. Dyer for a large number of helpful suggestions leading to the 
development of this generalized notion of root systems. 

The main body of this paper is organized into 2 sections, namely. 
Section |5] and Section Ei In Section |5] we develop a notion of root sys- 
tems applicable to a large family of groups that are generated only by 
involutions, and we investigate when these root systems may decom- 
pose into disjoint unions of the so-called positive roots and the so-called 
negative roots, and we prove that such groups are Coxeter groups only 
if such decompositions take place (Theorem 12.31 and Theorem 12. 8p . In 
Section [3] we prove that the notion of root systems in [L2\ and [13] 
applies to all the reflection subgroups of any Coxeter group. In partic- 
ular, we give a geometric characterization of the roots that correspond 
to the Coxeter generators of reflection subgroups (Proposition 13 . 151 and 
Proposition I3.22|) . and we show that these characterizations are pre- 
cisely those allowing these roots to be the simple roots for the root 
systems of such reflection subgroups within the root systems of the 
respective over-groups. 

Notation. If A is a subset of a real vector space then we define the 
positive linear cone of A, denoted PLC(y4), to be the set 

{ Catt I Cq > for all a G A, and Ca' > for some a' & A }. 

Furthermore, we define —A := { —v \ v E A}. Also, if S is a subset of 
a group G then {B) denotes the subgroup of G generated by B. 

2. Decomposition of Root Systems and Coxeter Datum 

Let Vi and V2 be vector spaces over the real field M equipped with 
a bilinear pairing ( , ) : Vi x V2 — )■ M. Let S be an indexing set, and 
suppose that Hi := { | s G 5 } C Vi and II2 '■= { f^s \ s E S } C V2 
are both in bijective correspondence with S. Further, suppose that Hi 
and 112 satisfy the following conditions: 

(Dl) {as, Ps) = 1 for all s e S; 

(D2) (i) ^ PLC(ni) and ^ PLC(n2). 

(ii) as i PLC(ni \ {as}) and /3, ^ PLC(n2 \ {Z?.}) for each 
s e S. 
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Observe that condition (D2) (i) implies that ^ PLC(— Hi \ {— tts}) 
and ^ PLC(— 112 \ {—/3s}) for each s e S. We remark that there 
do exist examples for which Hi (resp. 112) is linearly dependent, in 
which case necessarily some (resp. /Sg) will be expressible as a linear 
combination of Hi \ {ag} (resp. 112 \ {Ps}) with coefficients of mixed 
signs. 

Definition 2.1. For s e S, define pi{s) E GL{Vi) and p2{s) G GL(V^2) 
by the rules 

Pi{s){x) := X - 2{x,/3s)as 

for all X E Vi, and 

P2{s){y) ■.= y- 2{as,y)/3s 
for all y G V2. Further, wc define, for each i G { 1, 2 }: 



and 



Ri 


:= {pi{s) \ seS}; 




■■= {R^)■, 




:= W.Hf, 




:= $,nPLC(n,); 




:= -$+. 



For each i G {1,2}, and for each s G 5, we call Pi{s) the reflections 
corresponding to s in V^. We call the roo^ system for the VFe?// 
^Towp realized in Vi, and we call Hi the set of simple roots in 
Furthermore, we call the set of positive roots in $i and $7 the set 
of negative roots in 

Remcirk 2.2. For each i G {1, 2} and each s G note that pi{s) is an 
involution with a — 1-eigenvector of multiplicity 1. Furthermore, it is 
a consequence of condition (D2) that n — 0. Use (±) to denote 
disjoint unions, we have: 

Theorem 2.3. Given conditions (Dl) and (D2), the following are 
equivalent: 

(i) $1 = $+ y 

(ii) $2 = W ^2 • 

(iii) For all s,t E S the following three conditions are satisfied: 
(D3) {as, Pt) < and {at, Ps) < whenever s ^ t. 

(D4) {as, (5t) = z/ and only if {at, f3s) = 0. 
(D5) Either {ag, fit) {at, f3s) — cos^ ^ for some integer rUst > 2, 
or e/se {ag, Pt){o(t, Ps) > 1- 

It is a consequence of this theorem that if any of the equivalent 
conditions in it is satisfied then Wi and 1^2 are isomorphic Coxeter 
groups. To prove this theorem we shall need a few technical results 
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first. These are essentially taken from [LQl, and for completeness, the 
relevant proofs are included here. 

Let =2/ be a commutative M-algebra, let g^/^ and X be units of jz/, 
and let 7 G M. Define A, 5 to be 2 x 2 matrices over ^ given by 



A 



-1 27gi/2^ 
q 



B 







27gV2x-i 



It is easily proved by induction on n G N that 



B{ABy 
A{BAy 
{BAY 



-q P2n+1 
-q 2p2nX 



-q P2n^l 

-q"-\p2nX-' 



-q^^2p2nX 
g>2n+l 



and 



{ABy 



q P2n+l 



-g"-2p2„X 

-g>2n-l 

where J9„ G R (n G { — 1 } U N) are defined recursively by 

P~i = -1, Po = 0, = 27p„ - pn-i {neN). 

The solutions of the recurrence equation (12. 5p is 

n 7=1 
(-l)"+in 7=-l 
sinh n6 



(2.1) 
(2.2) 
(2.3) 

(2.4) 
(2.5) 



Pn 



sinh 9 
sinnO 
sin 



(where 6 = cosh^ 7) I7I > 1 
(where 6 = cos~"'^7) I7I < 1. 



(2.6) 



Proposition 2.4. ([TOl Lemma 2.2]) Keep all the above notation. 

(i) Conditions (1) and (2) below are equivalent: 

(1) PnPn+i > for all n G N; 

(2) 7 G {cos^ I m G N,m > 2} U [l,oo). 

(ii) If 'J = cos ^ for some k,m eN with < k < m then the matrices 
A and B satisfies the equation 

ABA ■■■ = BAB ■ ■ ■ 

where there are m factors on each side. 

(Hi) If q = 1 then the matrix AB has order m if 'j = cos ^ for some 
k,m E N with < k < m and gcd(m, k) = 1, and the matrix AB has 
infinite order otherwise. 
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Proof, (i): First assume that (1) holds. Observe that fl2.5p yields that 
Pi = 1 and p2 = 27, hence 7 > 0. Since (2) obviously holds if 7 > 1, 
we may assume that < 7 < 1. Choose 6 so that < < | and 
cos6' = 7, and let m be the largest integer such that 

<e <2e < ■■■ <me <7r. 

Note that m > 2. Now if m6 ^ 71 then vr < (m + 1)6 < 27r, and in 
view of (O we have = ^ > 0, whereas p^+i = ^i^^^til^ < 0, 
contradicting (1). Hence m6 = it and 7 = cos ^ for some integer 
m > 2, whence (2) holds as desired. Conversely, if (2) holds then it 
follows immediately from ( 12. 6 p that (1) holds. 

(ii) : If m = 2r is even then our task is to prove that {ABY = {BAy. 
It follows from (12.61) that Pn = ^^^^^^0^^ and hence, P2r+i = (—1)'^ and 
P2^_^ = while p2r = 0. Then it follows from ([23]) and (123]) 
that {ABY = {BAy. 

If m = 2r+l is odd then our task is to prove that B{ABy = A{BAy. 
In this case we find from (12. 6p that p2r+i = 0, while p2r+2 = and 
P2r = (—1)'^^^, and then the required result follows immediately from 
(O and m\f . 

(iii) : If 7 = cos ^ then it follows immediately from (ii) above that 
(y4i?)™ = 1, because A'^ = B"^ = 1 when q = I. If < n < m and 
gcd(A;,m) = 1, then ([22]) yields that p„ = '''^j;^^^!^^ ^ 0, and it then 
follows from (12. 4 p that (Ai?)" 7^ 1, proving that AB has order m. On 
the other hand, if 7 is of any other form then it follows from (12. 6p that 
Pn ^ for all integer n > 0. Then it is clear from (12. 4p that {ABy 7^ 1 
for all such n, proving that AB has infinite order. □ 

Now we are ready to prove Theorem 12.31 

Proof of Theorem \2.3[ We give a proof that (i) is equivalent to (iii). An 
entirely similar argument shows that (ii) and (iii) are also equivalent. 

First we show that (iii) implies (i). Given conditions (D3), (D4) and 
(D5) of the present paper we observe that ^ := ( S*, Vi, V2, Hi, 112, (, ) ) 
forms a Coxeter datum in the sense of [H], and hence (i) follows im- 
mediately from Lemma 3.2 of [Hj. 

Conversely, suppose that $1 = <I>^ l±l . We first prove that (D3) 
holds. Let s,t E S he distinct. By definition we have 

pi{t)as = as - 2{as, f3t)at. (2.7) 
The condition $1 = l±l implies that either 

Pi{t)as = ^^CrOr, where all > 0, (2.8) 

r€S 

or else 

Pi{t)as = —Crttr, where all > 0. (2.9) 
res 
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The following argument involving inspecting the coefficients rules out 
the possibihty of ( I2.9p . Indeed, in view of ( 12 .Zp we would have from 
dM} that 

(l + cj+ Crar = {2{as, Pt) - ct)at. 

r&S\{s,t\ 

Now if 2{as,f3t) — q > then we have a contradiction to (D2), since 
then at G PLC(ni \ {at}); whereas if 2{as,(3t) — Q < then we again 
have a contradiction to (D2), since then G PLC(ni). Thus (12 .Sp must 
be the case, and in view of (12 .yp we have 

(1 - Cs)as = {2{as, Pt) + ct)at + ^ Cr-a^. 

reS\{s,t} 

Suppose for a contradiction that {as, Pt) > 0. Then 2{as, f3t) + q > 0. 
Now if 1 — Cs > then we have a contradiction to condition (D2), since 
as G PLC(ni \ {os}); whereas if 1 — < then we again have a 
contradiction to (D2), since G PLC (Hi). Thus it follows from these 
contradictions that {as, Pt) < 0, and interchange the roles of s and t, 
we see that {at, (5s) < 0, whence (D3) holds. 

Next, suppose that further {as,(3t) = 0, and we prove that (D4) 
holds. Observe that 

pi(t)pi{s)at = pi(t){pi{s)at) = -at - 2{at,f3s)as + 4:{at, (3s){as, f3t)at 

= -at - 2{at,f3s)as. 
Again the assumption that $1 = W implies that either 

— at — 2{at, f3s)as = c^ar, where all > 0, (2-10) 

res 

or else 

— at — 2{at,(3s)as = — c^a^, where all > 0. (2-11) 

res 

A similar argument involving inspecting the coefficients together with 

(D2) make it possible to conclude that only (12.111) is possible. Hence 

{-2{at, f3s) + Cs)as + ^ c,a, = (1 - cja^. (2.12) 

reS\{s,t} 

Observe that (D3) just prove above yields that —{at,f3s) > 0. Now 
if 1 — Q < then we will have a contradiction to (D2), since then 
G PLC (Hi); whereas if 1 — q > then we again have a contradiction 
to (D2), since then at G PLC(ni \ {aj). Thus q = 1, and then fl2T2|) 
implies, in view of (D2) and (D3), that {at, (3s) = = Cs (and = 
for all r G 5" \ {s, t}). Interchange the roles of s and t we deduce that 
{at, (is) = implies that {as, (it) = 0, whence (D4) holds. 

To prove that (D5) holds, we may assume that {as, (it) {(^t, (is) 7^ 0, 
for otherwise {as, (3t) {at, (is) = cos^ |, trivially satisfying (D5). We let 
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■c/, 7, q, X, pn, A and B be as defined before Proposition 12.41 If we 
set 

^ = M; 
g = l; 

7 = a/ {as,l3t){at,l3s); 

and 

^ ^ -(at,/3s) 

A/(as,/3t)(at,/3s)' 

then it is readily checked that A and -B are the matrices representing the 
actions of pi(s) and pi(t) respectively, on the ({ pi{s), pi(t) })-invariant 
subspace Ra^ + M.at. It follows from (12.11) to (12. 4p and a similar ar- 
gument involving inspecting the coefficients as used above that the 
requirement 

({pi(s),pi(t) })as U ({pi(s),pi(t) })at C $+ W <l>r 

is equivalent to PnPn+i > for all n G N. By Proposition 12. 4[ this later 
condition is, in turn, equivalent to 

{as, (3t) {at, /3s) E { cos^ — | m G N>2 } U [l,oo), 

m 

whence (D5) holds, finally establishing that (i) implies (iii). □ 

Notation 2.5. For Wi G Wi (for each i G {1, 2}), let ord(wj) denote the 
order of Wi in Wi. For those s,t E S with {as, f3t){at, f3s) > 1, extend 
the definition of rrist (given in Theorem 12. 3p by setting rrist = oo. 

Proposition 2.6. Suppose that one of the (equivalent) statements of 
Theorem \2.3\ is satisfied. Then ord(pj(s)pj(t)) = rUst. 

Proof. If one of the (equivalent) statements of Theorem 12.31 is satis- 
fied, then ^ := { S,Vi,V2,Ili,Il2, {,) ) forms a Coxeter datum in the 
sense of [H], and thus the required result follows immediately from 
Proposition 2.8 of [14J. □ 

We point out that a Coxeter datum in the sense of automatically 
satisfies the conditions (Dl) to (D5) of the present paper. Indeed, the 
only possible difference of these two formulations is that in (D2) of 
the present paper we require a seemingly extra condition that as ^ 
PLC(ni \ {as}) and /3, ^ PLC(n2 \ {/?.}) for each s e S. However, 
it can be checked that this condition is an immediate consequence of 
(CI), (C2) and (C5) of a Coxeter datum in the sense of [Hj (in fact, 
this is just p3l Lemma 2.5]). Thus we have: 

Proposition 2.7. The following are equivalent: 

(i) ^ := ( 5, Vi, V2, Hi, 112, ( , )) satisfies one of the (equivalent) 
statements of Theorem \2.3\ : 
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(ii) := ( S', Vi, V2, Hi, 112, {, )) is a Coxeter datum in the sense of 

□ 

Next we have a result which enables us to give a characterization of 
Coxeter groups, among a large family of linear groups that are gener- 
ated by involutions, in terms of their root systems: 

Theorem 2.8. Let S , Hi and II2 be the same as at the beginning of 
this section, and let Ri, Wi, $1, R2, W2 and $2 be as in Definition \2.1\ 
Let {W, R) be a Coxeter system in the sense of p] or [21], with W being 
an abstract group generated by a set of involutions R := {vg \ s E S } 
subject only to the condition that for s,t E S the order ofrgTi is either 
equal to m if {ag, f3t){<^t, Ps) = cos^(7r/m), or else equal to infinity. 
Then $1 = ^ , or equivalently , $2 = $^ W $^ only if there exist 
isomorphisms fi: W Wi and f2 - W W2 such that fi{rs) = Pi{s) 
and f2{js) = P2{s) for all s E S. 

Proof. Follows immediately from Proposition 12.71 above and [TU The- 
orem 2.10]. □ 

Remark 2.9. Theorem 12.81 shows that if $1 = $^ W or equiva- 
lently, $2 = l±l ^2 then (Wi,Ri) and (1^2,-^2) are Coxeter systems 
isomorphic to {W, R). It is well known in the literature that all Coxeter 
groups have root systems decomposable into a disjoint union of posi- 
tive roots and negative roots ( [H Proposition 4.2.5] or [211 §5-4], for 
example). Furthermore, given an arbitrary Coxeter system {W',R'), 
it follows from and [14j that we could associate a Coxeter datum 
^' := {S',VI,VIU[,U'2,{, Y) to {W',R'), such that the paired root 
systems and $2 arising from this particular Coxeter datum admit 
decompositions ^[ = ^'^ l±l $'f and $2 = "^2^ ^ ^'2 ■ These facts com- 
bined with Theorem 12.81 yield that if a linear group is generated by 
involutions, then it is a Coxeter group if and only if it has a root sys- 
tem decomposable into a disjoint union of positive roots and negative 
roots. 

Let W and R be as in Theorem 12. 8[ we call (W, R) the abstract 
Coxeter system corresponding to ^ with W being the corresponding 
abstract Coxeter group. We see immediately from the above theorem 
that /i and /2 give rise to faithful ly-actions on Vi and V2 in the natural 
way with wx := and wy := (/2('w))(y) for all w G W, x E Vi 

and y G V2. 

To close this section we include the following useful result taken from 

m- 

Lemma 2.10. (i) (, ) is W -invariant, that is, {wx,wy) = {x,y) for 
all w G W , x E Vi and y E V2. 

(ii) There exists a W-equivariant bisection 0: $1 — ?■ $2 satisfying 
= Ps for all s E S. 
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(in) Let (f) be as in (ii) above, and letx,x' G $i. Then = 

if and only if {x', (f){x)) = 0. 



Proof, (i): Lemma 2.13 of [H]. 

(ii) : Proposition 3.18 of [H]. 

(iii) : Corollary 3.25 of [S]. 



□ 



For the rest of this paper, the notation will be fixed for the W- 
equivariant bijection in Lemma [2.101 (iii). 

3. Reflection Subgroups and Canonical Generators in 

CoxETER Groups 

Given a Coxeter group W and its Coxeter generators R, a subgroup 
W of W is called a reflection subgroup if W is generated by those 
elements of the form wrw~^ (where w E W and r G i?). It is well 
known that W is a Coxeter group, and consequently the notion of 
a Coxeter datum as in the previous section applies to W. In this 
section we study the paired root systems for W as a subsets of the 
paired root systems for W. In the spirit of the previous section, our 
investigation of the paired root systems for W is based on a Coxeter 
datum closely related to the Coxeter datum for the over group W. In 
particular, we show that the Coxeter generators of W are characterize 
by this Coxeter datum In addition to obtaining certain geometric 
insights of reflection subgroups of Coxeter groups, these investigations 
also establish the fact that the method of constructing paired root 
systems via Coxeter data applies to paired root systems for reflection 
subgroups of a Coxeter group, either on their own or as subsets of the 
paired root systems of the over group. 

Suppose that := {S, Vi, V2, Hi, 112, ( , ) ) satisfies conditions (Dl) 
to (D5) of Section |2] inclusive (or in view of Proposition \2.7\ we could 
equivalently suppose that is a Coxeter datum in the sense of [Ti]). 
Let {W, R) be the abstract Coxeter system associated to the Coxeter 
datum and keep all the notation of the previous section. 

Let T := IJwgH/ wRw~^, and call it the set of reflections in W . For 
s G 5* and w G W ^ observe that for each x E Vi and y G V2, Lemma [2.10l 
yields that 

wrgW'^x = wiw^^x — 2{w^^x, f3s)as) = x — 2{w^^x, f3s)was 

= X — 2{x, (j){was))was, (3.1) 

and 

wTsW^^y = w{w~'^y - 2(0^, w'^y)f3s) = y - 2{was, y)wf3s 

= y-2{<P'\wl3s),y)w(3s. (3.2) 

Now suppose that a G $1 and /3 G $2 are arbitrary. Then a = Wiag 
and (3 = W2A for some Wi,W2 G W and s,t E S. It follows from (13. ip 
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and fl3.2p that we can unambiguously define Tq,, rp G T, the reflection 
corresponding to a and (3 respectively, by 

Ta = Tw^as ■= WirsW^^, (3.3) 

and 

rp = r^2f3t ■= W2rtW2^, (3.4) 

with 

= X — 2{x, (f){a))a 

for all X E Vi and 

rpy = y-2{r\P),y)/3 

for all y E V2. 

Definition 3.1. (i) A subgroup W of W is called a reflection sub- 
group ifW'={W'nT). 

(ii) For each i G {1, 2}, a subset $^ of $j is called a root subsystem if 
r^y G $^ whenever x,y E 

(iii) If is a reflection subgroup, set ^i(W') := {x e \ e W } 
for each i G 1,2. 

Lemma 3.2. Lei W be a reflection subgroup of W . Then for each 
IE {1,2} 

W'<^i{W') = <^i{W'). 

Proof. We prove that W'^i{W') = ^i{W') here, and we stress that the 
other half follows in the same way. Let w E W. By definition, we have 
w = tit2 ■■■in where ^1,^2, ■ ■ ■ ,tn e 11" fl T. Now let x E ^i{W') be 
arbitrary. Then E W, and hence rt„x = tn^xtn £ W ■, which in turn 
yields that tnX E ^liyV'). Then it follows that t„-it„a; G $i(W^') and 
so on. Thus wx = ti ■ ■ -tnX E $i(ll^'). Since x E $i(ll^') is arbitrary, 
it follows that w^i{W') C $1(11^'). Finally, replacing w eW hy 
we see that <l>i(ll^') C w<l>i(ll^')- □ 

Remark 3.3. Let W be a reflection subgroup. For each i E {1,2}, 
it follows from the above lemma that $j(ll^') is a root subsystem of 
and we call it the root subsystem corresponding to W. It is easily 
seen that there is a bijective correspondence between the set of reflec- 
tion subgroups W of 11^ and the set of root subsystems $• of W 
uniquely determines the corresponding root subsystem ^i{W')] and $^ 
uniquely determines the reflection subgroup 11^' := {{r^ \ x E $^ }). 

In fact, for a reflection subgroup W, we shall see that ^i(W') and 
^2{W) are the root systems for the Coxeter group W arising from 
a suitably chosen Coxeter datum. In order to do this, we need a few 
preparatory results first. 

Remark 3.4. For each i E {1,2}, it has been observed in [H] that 
non-trivial scalar multiple of an element of $j can still be an element 
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of $j (see the example immediately after [T?l Definition 3.1] and [HI 
Lemma 3.20]). Therefore, unlike in the classical setting of [21j, we do 
not have a bijection from T to either or 

Definition 3.5. For each i G {1,2}, define an equivalence relation ~j 
on $j as follows: if 2:1,-22 ^ ^i, then Zi ~j Z2 if and only if Zi and ^2 
are (non-zero) scalar multiples of each other. For each z G write ^ 
for the equivalence class containing z and write $j = { ? | 2; G }. 

Remark 3.6. Observe that W has a natural action on $j (for each 
i G {1, 2}) given by w'z = wz for a\\ w E W and 2; G Furthermore, 
given G the corresponding reflections and r^/ are equal if 
and only if 2 = 2'. 

Definition 3.7. For i G {1, 2}, and for each w G W, define 
Ni{w) = {z \ z e and wz G $7 }. 

Note that for w G W, the set Ni{w) {i = 1,2) can be alternatively 
characterized as { ^ | z G $7 and G }. Hence 2^ G Ni{w) if and 
only if precisely one element of the set {z,wz} is in 

Notation 3.8. Let £ : 14^ — )■ N denote the length function with respect 
to {W, R), that is, for w eW, 

i{w) = min{ 77, G N I w = rir2 ■ ■ ■ r„, where ^i, r2, ■ ■ ■ , r„ G i? }. 

A mild generalization of the techniques used in ([211 §5-6]) then yields 
the following connection between the length function and the functions 

Ni and N2: 

Lemma 3.9. ([H Lemma 3.8]) (i) A^i(r,) = {a,} and A/'2(r,) = {^S,} 
/or a// s G S". 

(ii) Let w & W. Then Ni{w) and N2{w) both have cardinality i{w). 

(iii) Let wi, W2 E W and let + denote set symmetric difference. Then 
Ni{wiW2) = W2'^Ni{wi) + Ni{w2) for each i G {1, 2}. □ 

The last lemma enables us to deduce the following generalization of 
[Tit Lemma 3.2 (ii)]: 

Proposition 3.10. For each i G {1,2}, let w e W and x G . If 

liwr^) > i{w) then wx G ^f, whereas if liwr^) < i{w) then wx G . 

Proof. We prove the statement that ilwr^) > i{w) if and only if wx is 
positive in the case a; G $1, and again we stress that a similar argument 
also shows the desired result holds in $2. 

Observe that the second statement follows from the first, applied to 
wr^ in place of w: indeed if i^wr^) < i{w) then i{{wrx)ra;) > i^wr^), 
forcing (wr^)x = w^r^x) = —wx G ^f, that is, wx G 

Now we prove the first statement in $1. Proceed by induction on 
i{w), the case i{w) = being trivial. If i{w) > 0, then there exists 
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s E S with lirgw) = i{w) — 1, and hence 

i{{rsw)r^) = l{r-s{wr^)) > E{wr^) - 1 > i{w) - 1 = i{rsw). 

Then the inductive hypothesis yields that {rsw)x G . Suppose for 
a contradiction that wx G $f. Then wx G Ni{rs) and Lemma [3.91 (i) 
yields that wx = — Aa^ for some A > 0. But then rgWX = Aa^, and 
hence (rsw)r^(rsw)"^ = by calculations similar to (13.31) and (13. 4p . 
But this yields that wr^ = TgW, contradicting i^wr^) > i{w) > i{rsw), 
as desired. □ 

Definition 3.11. For each w G W, define 

N{w) ■={teT\ i{wt) < i{w) }. 

If t G T then t = wVsW^^ for some w ^ W and s G S*, and hence 
it follows from calculations like (13. 3 p and (13. 4p that t = r^^^ = r^p^. 
This combined with Proposition 13.101 give us: 

Proposition 3.12. Let w eW . Then 

N{w) = {r, \ xeNiiw)} 
for each i G {1, 2}. □ 

Definition 3.13. Suppose that W is a reflection subgroup. Then we 
define 

S{W') := { t G T I N{t) f^W' = {t} } 

and 

^i{W') ■= {x G $+ I G S{W') } 

for each i G {1, 2}. 

For a reflection subgroup W , the set S{W') is called the canonical 
generators of W in [10], and it is well known that {W' , S{W')) is a 
Coxeter system. Indeed, we have: 

Lemma 3.14. [10] Let W be a reflection subgroup ofW. 

(i) liteW'nT, then there exist m G N and to, ■ ■ ■ , e S{W') such 
that t = tffi ■ ■ ■ titoti • • • tm- 

(ii) {W, S{W')) is a Coxeter system. 

Proof, (i): [IHl Lemma (1.7) (ii)]. 

(ii)]: [H Theorem (1.8) (i)]. □ 

For a reflection subgroup W, we will show that Ai{W') and A2{W') 
can be characterized in terms of a suitably defined Coxeter datum. 
Before we could prove this, we need a number of simple observations. 

Observe that for a reflection subgroup W we can equivalently define 
Ai{W') by requiring 

\{W') := { a; G I N,{r,) n ^^W^) = {x} }. (3.5) 
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Suppose that A'^ C and A'2 C $^ are two sets of roots with 
4>{A[) = A2 (where is as in Lemma [2. 101 (iii))- Furthermore, suppose 
that A[ and A2 satisfy the following: 

(i') {x,(j){x')) < 0, for all distinct x,x' e A[] 

(ii') {x,(j){x')){x',(j){x)) e {cos^(7r/m) \ m eN,m>2}U [I, 00), for 
all X, x' G A']^ with 7^ rj,'. 

It follows from Lemma 12.101 that 

(x, = 1, for all X G a;. (3.6) 

Since A; C PLC(ni) and A'2 C PLC(n2), it follows that ^ PLC(A;) 
and ^ PLC(A2). Also it can be readily checked from (i'), (ii') and 
dSSD that X i PLC(A; \ {x}) and 0(a;) ^ PLC(A'2 \ {<p{x)]) for all 
X G A'^. Furthermore, Lemma 12.101 (iii) ensures that {x,(f){x')) = 
whenever {x',(f){x)) = for all x,x' G A'^ C <J)]^, Thus A'^ and A2 
satisfy conditions (Dl) to (D5) of the present paper inclusive. If we let 
S' be an indexing set for both A'^ and Ag then 

^' := ( S', span(A;), span(A^), A^, A'^, (,)'), 

(where ( , )' denotes the restriction of ( , ) to span(A']^) x span(A2)) con- 
stitutes a Coxeter datum in the sense of |I4j. Now if we let R' := {r^ \ 
X G A'^ }(= {vy \ y e }), and set W = {R'), then it is clear that 
W is a reflection subgroup of W. Furthermore, it follows from The- 
orem 12.81 that {W, R') forms a Coxeter system. Then upon applying 
Lemma [33] and (13. 5p to and W we may conclude that S(W') = R! 

and consequently Ai{W') = A[ and A2(W') = Ag. Summing up, we 
have: 

Proposition 3.15. Suppose that A[ C and A2 C $^ such that 
(Al) 0(A;) = A'2; 

(A2) {x, (j){x')) < 0, for all distinct x, x' G A'^; 

(A3) {x,(j){x')){x',(f){x)) G {cos^(7r/m) |mGN,m>2}U [l,oo), for 
all X, x' G A[ with 7^ r^,/. 

Then W = ({ | x G A'^ }) is a reflection subgroup of W with A'^^ = 
AAW') and A^ = A^'). □ 

It turns out that the converse of Proposition [3?T5] is also true, namely: 
if W is a reflection subgroup of W, and if x,x' G Ai{W') with 7^ 
r^t, then conditions (A2) and (A3) of Proposition 13.151 must be sat- 
isfied. Since Lemma [2.101 (iii) ensures that (x,0(x')) = if and only 
if (x'0(x)) = 0, it follows from this assertion and a quick argument 
similar to the one used immediately after (13. 6p that representative ele- 
ments from Ai{W') and A2{W') can be used to form a Coxeter datum 
for W. Hence this assertion and Proposition 13.151 together yield that 
for a reflection subgroup W, the corresponding Ai{W') {i = 1,2) can 
be characterized by a suitable Coxeter datum. We devote the rest of 
this section to a proof of this assertion. 
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Lemma 3.16. Let W be a reflection subgroup ofW. 

(i) For each i G {1,2}, let x e \ Then Ai{r^W'r^) = 

(ii) For each 2 G {1,2}, ^^{W') = W'Ai{W'). 

Proof, (i): It is readily checked that r^i{W') = ^i{rW'r) for all r G 
T. Since x G \ <I>,{W'), it follows that e R\ W. Let y G Ai{W') 
be arbitrary. Then 

(by (E3D and ([S3D) 

= {r,Ni{r,ry) + N,{r.,)) n $.Mi^r,) 
(by Lemma 13.91 (iii)) 

= {r,ryN,{r,) + r,N,{ry) + A/,,(r,)) 

(again by Lemma ES] (iii)) 

= r^{{ryN,{r,) + N,{ry) + Ni{r,)) 

= r,{{ry{x} + N,{ry) + {x}) H $^(1^)) 
(by Lemma 13191 (i)) 

= r,.(iV,(r,)nC(W^)) 

(since {x),ryX ^ ^i{W')) 

= {€y} 

(since y G Ai(l^'))- 

Hence r^y G Aj(r2;iyra;). This proves that rxAi{W') C Aj(r^iyra;). 
But X G ni\r^$i(iy), so the above yields that rr,Ai{r^W'rr,) C Ai{W') 
proving the desired result. 

(ii): Since Ai{W') C <!>i{W') for each i G {1,2}, it follows from 
LemmaOthat W'Ai{W') C $^(1^')- 

Conversely if x G $i(iy) then r^. G W^' fl T. By Lemma EUl (i), 
there are xq, xi, ■ ■ ■ ,Xm & Ai(W') {m G N) such that 



Calculations similar to those of fl3.3p and (13. 4p enable us to conclude 
that Ax = Tj.^ ■ ■ -r^iXo G W'^^iiW') for some (nonzero) scalar A. Now 
since ^Xq = {j'^^ - ■ ■rx^)~^x G $i, it follows that ^xq G AiiW') and 
hence x = r^^ ■ ■ ■ r (t^^o ) G W'Ai{W') as required. □ 
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Definition 3.17. Let W be a reflection subgroup of W, and let iw' '■ 
W ^Nhe the length function on (W', S(W')) defined by 

iw'i'u^) = niin{ n G N | = ri ■ ■ ■ r„, where rj G S(W') }. 

li w = ri ■ ■ ■ Tn E W (r, G S{W')) and n = iw'{u)) then ri ■ ■ - rn is 
called a reduced expression for w (with respect to S{W')). 

Lemma 3.18. Let W be a reflection subgroup. For each i G {1,2}, 

(i) Ni{r^) n = {x} for all x G /^i{W'); 

ill) for all wi eW and W2 G W 

Ni{wiW2) n ^dw^) = w^^Niiwi) n <C(W^)) + iNi{w2) n <Ca^)). 

Proof, (i) is just the definition of Aj(iy). 

(ii) Lemma [33] (iii) yields that Ni{wiW2) = w^^A^j(wi) + Ni{w2), 
and hence 

Ni{w,w2) n $;c^) = (^a'^^'il^i) n $^)) + (iv,(tf;2) n <C(i^)). 

Since W2 e W^' it follows from Lemma lO that W2'^^i{W') = ^i{W'). 
Thus W2^Ni{wi) n = W2'^(A/'i(w;i) n Sjjv')), giving us 

□ 

Lemma 3.19. Let W be a reflection subgroup. For each i G {1,2} 
and all w G W , we have 

(i) \Ni{w)n^i(W') \ = iw'iw). Furthermore, if w = ■ ■ ■rx„ (where 
xi, - ■ ■ ,Xn G Ai{W') ) is reduced with respect to {W' , S{W')) then 



where yj = (r^„ ■ ■ ■ r^.^^Xj for all j = 1, ■ ■ ■ , n. 

(ii) N,{w) n <C(W^) = {xe <C(W^) I iw'iwrx) < iw'iw)}. 

Proof (i): For each j G {1, ■ ■ ■ , n}, set tj = r^.„ ■ ■ ■ rx^^^r^T^j^^ ■ ■ ■ r^^ 
that is, tj = Ty. . If tj = tk where j > k then 



N,{w)n<l>,{W') = {yi,---yn} 



w = r. 



= r. 



■ ■ - r. 



■ ■ • r. 




= r. 



■ ■ ■ r. 



■ ■ ■ r. 



• • • r. 



contradicting iw'iw) = n. Hence the tj's are all distinct and conse- 
quently all the jjj^s are all distinct. Now by repeated application of 
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Lemma [3.181 (ii), for each i G {1, 2} we have 

Ni{w) n ^AW^) 
= {N,{r,„ n <h{w^)) + r,„(iv,(r„_i) n ^jyv^)) + ■■■ 

= {fn} + {yZ^i} + ■■■ + {yi} 
= {m,--- Sn} 

and consequently \Ni{w) fl = iw'iw). 

(ii): Let w = r^^- ■ -r^^ be a reduced expression for w G W with 
respect to S{W') (xi,--- , x„ G Ai{W')). Then for each i G {1,2}, 
Part (i) above yields that 

where yj = {r^^ ■ ■ ■ rx^^Jxj, for all j G {1, ■ ■ ■ ,n}. Now for each such j, 

^'Vj ~ '^'Xn ' Xj + i' Xj' Xj + i ' Xn — 'xi ' Xj-i' Xj + i 'Xn 

and so iw' {"wry^ ) < n — 1 < iw' (w). Hence if ir G A^j (w) fl ^i{W') , then 
£w'{wrj,) < iw'iw). 

Conversely, suppose that x G ^i(W') fl $^ and x ^ Ni{w). We are 
done if we could show that then iwi^wr^) > i]v/{w). Observe that 

the given choice of x implies that x G Ni{rx) fl ^i{W'), furthermore, 

X i rx{Ni{w) n CCW^))- Therefore 

X G r^iNiiw) n $^)) + (Niir,) n $^)) = Ni{wr,) f] 
and by what has just been proved, this implies that 
iw'iw) = ^w'{{u!rx)rx) < iw'iwr^), 

as desired. 

□ 

The following is a mild generalization of [101 Lemma 3.2]: 

Lemma 3.20. Let W be a reflection subgroup. For each i G {1,2}, 
let x,y E Ai(W') such that 7^ Vy. Let n = ord(r^.ry). Then for 
< m < n 

; • ■ryTn.ry X G and • • -r^Tyrn^ y G 

m factors m factors 

Proof. It is easily checked that when < m < n we have 

iwii ^ ■ -ryTxry yx) = m + 1 > m = £vF' ' 

m factors m factors 
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as well as 

^wii ^ ■ -rxryr^ yy) = m + 1 > m = iw'i^^^^VVj)- 

m factors m factors 

Hence the desired result follows immediately from Lemma 13. 191 □ 

In fact we can refine Lemma 13.201 with the following generalization 
of [ini Lemma 3.3]: 

Lemma 3.21. Let W be a reflection subgroup. For each i G {1,2}, 
let x,y E Ai(W') with ^ Vy. Let n = ord(r^ry), and let Cm,dm,c'^ 
and d'^ be constants such that 

(■ ■■ryr^ry) x = CmX + dmV and (■ ■■r^ryr^) y = c'^x + d'^y. 

m factors m factors 

Then Cm > 0, dm > 0, > and (i^ > whenever m < n. 

Proof. By symmetry, it will suffice to prove that rfm > and d'^ > 0. 
The proof of this will be based on an induction on i{rx). 

Suppose first that i{rx) = 1. Then Xx G Ilj for some A > 0. Write 
y = -^2^ where > for all z G Ilj. In fact, A^„ > for some 

Zq E {x}, since otherwise we would have y G Mx and so = Ty. 
Now for < m < n. Lemma 13.201 yields that 

( ■ ■ ■ ryr^ry )x = CmX + ^ dmX^z G 

m factors ^Gllj 

That is 

CmX + dmC^ Xzz) = fi^z, where Hz > 0, for all z G Ilj. 
zgiii zeiii 

Now if dm ^ then the above yields that 

(Cm — fJ'x)x = {fiza ~ C^mAzo)^0 + (A^z ^ dmXz)z, 

zGni\{x,zo} 

contradicting condition (D2). Therefore, dm > as required. Similarly 

d'm>0. 

Suppose inductively now that the result is true for refiection sub- 
groups W" of W and x', y' G Ai{W") with r^./ 7^ r^/ and £(r^') < i{rx) 
where £(r^) > 3. It is well know that there exists 2 G Ilj such that 
(-(TzTxTz) = i{rx) - 2. Then £{r^rz) < i{rx), and thus z G Ni{rj:). But 
since x G Ai{W') and x ^ z (since iir^) > 3), it follows that ^ W . 
Let W" = TzW'rz. Lemma EH] (i) yields that A^iW") = r,Aj(iy') 
and therefore r^x, r^y G Ai{W"). Now 

r{r,x) = TzTxTz and r(^r,y) = TzTyTz (3.7) 
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and hence OTd{r(^r,x)r(r,y)) = oid^r^ry) = n. Since (-{r^r.x)) = ^{r^) - 2, 
the inductive hypothesis gives 

' v ' 

m factors 

and 

' V ' 

m factors 

where dm, d'^ > for < m < n. Finally, by (13. 7p . the desired result 
follows on applying to both sides of the last two equations. □ 

Proposition 3.22. Let W be a reflection subgroup of W . Suppose 
that x,y E Ai(W') with ^ Vy. Let n = ord(rxTy) G {oo} U N. Then 

{x,<Piy))<0 

and 

{x, ^iy)){y, = cos^ ^ (n G M, n > 2) 

G [l,oo) {n = oo) 

Proof. Observe that since r^^j = ^ Vy = r^(^y), it follows that 
{x,y} and {(f){x) , (j){y)} are both linearly independent, and hence con- 
ditions (Dl) and (D2) are satisfied. Now let us set R" = i?2 •= 
{''^xy'Ty} and W[' = W2 := {{rx,ry}), and furthermore, := Wi{x,y}. 
Observe that consists of elements of the form ±(- ■ ■ ryrxry)x and 

m factors 

^i' ■ ■ f'xf'yf'x)y (where < m < ord(r^ry)). Lemma 13.211 then yields 

m factors 

that $'/ = l±l , and consequently Theorem 12.31 yields that 

{x, 0(y)) (y, 0(x)) = cos2 ^ (n G N, n > 2) 

G [1,00) (n = 00). 

□ 
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